Sonification Design and Music

library(gm)
library(music)
library(tidyverse)
library(fpp3)

## read in pre-processed data
load("data/monthly_weather")

Packages

We used the ggplot2 package (2016) in R (2021) for visuals, and the gm package to make the music (Mao 2024).

Splitting into parts

After a brief conversation with the choreographer, Josh Schneider, it was determined that having all three series (maximum, minimum, average) at once was a bit much for the full history of the data set. As an homage to our shared passion for musical theatre, we settled on a “three act” (three part) structure. As such, we need some cut points for the sonification. The first is when the all time high temperature for the entire history of Paso Robles was hit:

## New high for Paso Robles
change_date_part_1 <- which.max(monthly_weather$max);
monthly_weather[change_date_part_1,c("date", "max", "min")]

# A tibble: 1 × 3
  date         max   min
  <date>     <int> <int>
1 1933-08-01   117    43

Auditorially, we represent this change by splitting from only tracking the average to tracking both the maximum and the minimum.

For the second part, the year that climate scientists began building models predicting the increasing temperatures we have seen on a global scale, 1977. Auditorially, we represent this as adding back in the average, but also tracking every time a new seasonal extreme is reached, where we start the clock at 1977.

## 1977 the year exxon scientists knew about climate change (allegedly)
change_date_part_2 <- min(which(monthly_weather$year == 1977))
monthly_weather[change_date_part_2,]

# A tibble: 1 × 17
   year month   min  mean   max date       season season_color season_year
  <int> <chr> <int> <dbl> <int> <date>     <chr>  <chr>              <dbl>
1  1977 Jan      23  46.6    70 1977-01-01 Winter #2f77c3             1977
# ℹ 8 more variables: season_label <chr>, xmin <date>, xmax <date>,
#   seas_avg <dbl>, seas_max <int>, seas_min <int>, new_max <lgl>,
#   new_min <lgl>

Sonification method

An earlier project on sonifying five number summaries inspired the sonification design, which was in turn inspired by Flowers and Hauer (1993). Additional inspiration came from the work of Peres and Lane (2003) as well as the wonderful Sonification Handbook(2011), in particular, chapter 8.

Sonifcations are generated via the custom function data_to_sonif(), which maps the largest value in the data set to a pre-specified low note, and the highest value to a pre-specified low note, then maps values in between maintaining the spacing between points rounded to the 12 tone equal tempered scale. For example, see how we sonify the five number summary to one octave:

Visualization of the pitch mapping sonification method.

source("code/data_to_soniof_all.R")

Part 1: Just averages

The averages are sonified to a two octave range by mapping the lowest value to the low pitch of the range and the high note. As specified further below, the average will be played by a viola.

## sonify first and last parts separately
sonif_all_mean = c(
  data_to_sonif_all(monthly_weather$mean[1:(change_date_part_1-1)], low = 3, high = 4),
  data_to_sonif_all(monthly_weather$mean[change_date_part_1:(change_date_part_2-1)], low = 3, high = 4),
  data_to_sonif_all(monthly_weather$mean[change_date_part_2:nrow(monthly_weather)], low = 3, high = 4)
  )                                   
sonif_all_mean[change_date_part_1:(change_date_part_2-1)] <- NA


pattern = rep("eighth", times = length(sonif_all_mean))
## lines for avg and max 
line_avg <- Line(pitches = sonif_all_mean, durations = pattern, 
                 name = "Monthly Average")

Part 2: Highs and Lows

The variability of the monthly maximum temperatures is slightly higher than that of the monthly minimum (Range of max: {r}diff(range(monthly_weather$max, na.rm = T)), Range of min:{r}diff(range(monthly_weather$min, na.rm = T)); Standard deviation of max: 13.21, 8.54 ). To represent this difference auditorially, we allow the maximum to span 3 octaves (C5 to C8) and the minumum to span 2 octaves (C3 to C5). Note that the minimum has the same octave span as the mean, which has a range of {r}round(diff(range(monthly_weather$mean, na.rm = T)), 2) and a standard deviation of 9.21.

As defined below, the maximum is played by a celesta (a keyboard instrument) and the minimum is played by a bassoon.

sonif_all_max = c(
  data_to_sonif_all(monthly_weather$max[1:(change_date_part_1-1)], low = 5, high = 7),
  data_to_sonif_all(monthly_weather$max[change_date_part_1:(change_date_part_2-1)], low = 5, high = 7),
  data_to_sonif_all(monthly_weather$max[change_date_part_2:nrow(monthly_weather)], low = 5, high = 7)
  )   
sonif_all_max[1:change_date_part_1] <- NA

line_max <- Line(pitches = sonif_all_max, durations = pattern, 
                 name = "Monthly Max of Daily Maximum")

sonif_all_min = c(
  data_to_sonif_all(monthly_weather$min[1:(change_date_part_1-1)], low = 3, high = 4),
  data_to_sonif_all(monthly_weather$min[change_date_part_1:(change_date_part_2-1)], low = 3, high = 4),
  data_to_sonif_all(monthly_weather$min[change_date_part_2:nrow(monthly_weather)], low = 3, high = 4)
  )   
sonif_all_min[1:change_date_part_1] <- NA

line_min <- Line(pitches = sonif_all_min, durations = pattern, 
                 name = "Monthly Min of Daily Minimum")

Part 3: Exceedances

While it is an issue that temperatures on average are increasing, it is not audible on the scales played here since the change is very slow over time. However, the average is not the only statistic available to us, and in times of change we don’t necessarily want an estimate of just “typical” values.

One important tool for understanding how the extremes of a time series are changing over time are exceedance occurrances: tracking when a new high or a new low is hit.

If we just tracked new all time highs and lows, we would essentially only capture the worst winter/summer. Instead, we start the tracker at 1977 for reasons mentioned above,and track each season (Winter, Spring, Summer, Fall) so we see a sudden influx of “new” maximums and minumums as we begin the tracking, but then once we initialize we can then track when unusually high or low values occurr. I set the pitches at middle C for both to avoid the sonification becoming too busy.

As defined later, the exceedances occurrances are played with a viola to elevate the importance of exceedance occurrances as a statistic along with the average, which is also played by the viola.

sonif_all_new_max = rep(NA, times = length(sonif_all_mean))
sonif_all_new_max[monthly_weather$new_max] <- "C4"
sonif_all_new_min = rep(NA, times = length(sonif_all_mean))
sonif_all_new_min[monthly_weather$new_max] <- "C4"

pattern_seas <- rep("quarter", times = length(sonif_all_max))

line_max_new_seas <- Line(pitches = sonif_all_new_max, durations = pattern, name = "New Seasonal Maximum Reached")

line_min_new_seas <- Line(pitches = sonif_all_new_max, durations = pattern, name = "New Seasonal Minimum Reached")

Create the music!

Here is where the instrumentation is actually added, and the music is outputted as an .mp3 file and/or score.

The choice of 12/8 time signature means that each measure is 1 year (12 months), with 4 beats felt per measure representing the four seasons, where each month is part of a seasonal triplet.

music_all <- Music() + Meter(12, 8) + 
  line_max + line_avg +  line_min + line_max_new_seas + line_min_new_seas + 
     Instrument(9, 1) +  ## monthly max, celesta
     Instrument(42, 2)  + ## monthly average, viola
     Instrument(71, 3) +  ## monthly min, bassoon
     Instrument(42, 4) +  ## new high, viola
     Instrument(42, 5) +  ## new low, viola
  Tempo(120) 

gm::show(music_all, to = c("audio"))

# gm::export(music_all, "music_sync.mscz")
 gm::export(music_all, "mmmm_sonification_paso_robles.pdf")

Learning more

Contact Dr. Julia Schedler with questions or comments! Thanks for perceiving!

References

Flowers, John H., and Terry A. Hauer. 1993. “‘Sound’ Alternatives to Visual Graphics for Exploratory Data Analysis.” Behavior Research Methods, Instruments, & Computers 25 (2): 242–49. https://doi.org/10.3758/BF03204505.

Hermann, Thomas, Andrew Hunt, and John G. Neuhoff, eds. 2011. The Sonification Handbook. Berlin: Logos Verlag.

Mao, Renfei. 2024. “Gm: Create Music with Ease.” https://cran.r-project.org/web/packages/gm/index.html.

Peres, S Camille, and David M Lane. 2003. “SONIFICATION OF STATISTICAL GRAPHS.” In Proceedings of the 2003 International Conference on Auditory Display.

R Core Team. 2021. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.

Wickham, Hadley. 2016. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org.